This invention relates generally to detecting and handling failures in a clustered array of mass storage devices such as an array of disk drives.
A redundant array of inexpensive disks (RAID) (called a “RAID array”) is often selected as a mass storage for a computer system due to the array's ability to preserve data even if one of the disk drives of the array should fail. There are a number of RAID arrangements but most rely on redundancy to achieve a robust storage system. For example, RAID 1 systems may utilize a mirror disk drive for redundancy. In other RAID systems such as RAID 2-5 systems, data may be split, or stripped, across a plurality of disk drives such that if one disk drive fails, the data may still be recovered by using the information contained on the still working disk drives in the system. As an example, in a parity RAID system such as a RAID 2-5 system, if three disks are utilized to store data and associated parity information, if one disk fails, the data may be recovered from the still working two drives in the system. A system having a single disk drive may be considered a RAID 0 system even though the system provides no redundancy.
A RAID array may also be part of a cluster environment, an environment in which two or more file servers share one or more RAID arrays. Typically, for purposes of assuring data consistency, only one of these file servers accesses a particular RAID array at a time to modify data. In this manner, when granted exclusive access to the RAID array, a particular file server may perform read and write operations as necessary to modify data contained in the RAID array. After the particular file server finishes its access, then another file server may be granted exclusive access to modify data in a particular RAID array.
For purposes of establishing a logical-to-physical interface between the file servers and the RAID array, one or more RAID controllers typically are used. As examples of the various possible arrangements, a single RAID controller may be contained in the enclosure that houses the RAID array, or alternatively, each file server may have an internal RAID controller. In the latter case, each file server may have an internal RAID controller card that is plugged into a card connector slot of the file server. Alternatively, the server may have the RAID functionality contained on a main printed circuit board.
For the case where the file server has an internal RAID controller, the file server (“Server”) is described herein as accessing the RAID array. However, it is understood that in these cases, it is actually the RAID controller card, or the RAID controller circuits on the main printed circuit board, of the server that is accessing the RAID array.
Before a particular server accesses a RAID array, the file server that currently is accessing the RAID array is responsible for closing all open read and write transactions. Hence, under normal circumstances, whenever a file server is granted access to a RAID array, all data on the shared disk drives of the array are in a consistent state.
In a clustering environment where different storage controllers access the same disk, the cluster operating system needs to guarantee data coherency and failure tolerance. Thus, there is a need for better ways to control the distribution of access rights, and for recovering from network failures, in clustered RAID networks.